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ABSTRACT 

The passage comprehensibility of three diagnostic 
reading tests— -the Stanford Diagnostic Reading Test, the Spache 
Diagnostic Reading Scales, and the Durrell Analysis of Reading 
Di^f iculty-^was evaluated using the following text variables: content 
familiarity, vocabulary, the number of idea units in the text, the 
number of text-based inferences required to integrate ideas/ the 1 
existence of main idea statements,, and the logical sequence of ideas 
in the passage. Selections from the second r fourth and sixth reader 
levels were evaluated to identify passage characteristics across 

* reader levels and test inconsistencies. WithT regard to text 
variables, each selection was transformed into a series of 
propositions or ideas. Also, eight graduate students rated each 
selection for content familiarity. With regard to readability formula 
variables, a computer program was, used in which each passage was 
evaluated in terms of four readability formulas: The ARI , 
Flesch-Kincaid, Harr is-Jacobson , and ]the Dale-Chall. Results showed 

• that the three tests differed substantially in the number of ideas 
presented, the number of inferences necessary to integrate these 
ideas, and text organization features. The readability indices 
reflected only surface features of the text and did not reflect the 
meaning characteristics of the passage nor identify the variables 
integral for deriving meaning' from the text. (HOD) 
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^' An Analysis of Text Variables in Three Current Reading Diagnostic Tests 



Considerable research has been conducted examining the text variables 

which influence comprehension. Collectively thfsf research* has indicated th$t 

variables such as, content structure (Meyer, 1975), propositions (Kintsch & 

Keenan, 1973), and- implicit/explicit connectives whichestablish relationships 

between ideas in a text (Marshall & Glock, 1978-79; Irwin, 1980) can 

facilitiate or inhibit comprehension performance. In addition, the degree of 

prior knowledge the reader brings to the topic can increase or decrease text 

processing demands (Kintsch, 1979; Marr & Gormley, 1982) . These findings have 

i 

significance not only_ f or i nstruct i onal pur-p-ases^-b^t -a-1 so -for t-he as-ses$n?ent 
of. reading comprehension proficiency. If the purpose of- assessment i s to 
identify a student's comprehension ability, then text variables wtiich inhibit 
comprehension r>eed to be identified and systematically controlled when-test 
passages are constructed thereby permitting the diagnostician to distinguish 
the student's comprehension proficiency from the comprehensibility of the 

text. At present, authors of reading diagnostic tests have failed to 

. , r . # 

construct such passages thus confounding comprehension abi 1 ity and text ~*'\. 

comprehensibility, making. the diagnosis of reading comprehension problems a 

, * ■ ' * 

most difficult task. 2 " • ' 



' • ""' ' Text Variables 

"' ' .... * 2 
To*further illustrate thMs. point; this paper identifies several text 
variables which have not been controlled in. passages selected. from three 
commonly used reading diagnostic tests and discusses how these tes.t passages 
may, in fact, inhibit comprehension. • . . 

Readability Formulae ■ " 1 4 

Traditionally, readability formulae have been used as indices of the 
difficulty level of text; that is, as an index which predicts the number of 
questions a : chi Id .may. answer correctly from a particular passage. Many of 
these early formulae were validated on selections from the McCall Crabbs 
Standard Test Lessons t - Surface features of the text such as the rTumber of 
words in*a passage'or the number of sentences in a passage were used in a 
linear regression formula to predict the difficulty level of a passage (Klare, 
1974-75). Several proponents' of readability formulae recognize that these 
formulae are predictive indices only, and that they were not designed to eause 
ease or difficulty in reading text nor provide assurance of improving 
readability (Klare, 1974^75). Unfortunately these formulae are not being used 
as they were intended. In fact, many individuals use these formulae as 
guideltnes'for increasing or decreasing the reading level of a passage. All 
tuo~ TrequeTrtTy indtvi duals coristructj^^ss^ge3;ca-tcirTate"a^readao;i THy-Teve J l- 
after- the passage has been written, then make adjustments to the passage to 
match the text with the reading levels of the students for whom the passage 
was intended. As a_ result, problems related to text comprehensibi.lity occur. 
When an original text is adapted to decrease readability, complex sentences 

i *■ • 

are frequently divided into simple sentences and* vocabulary is simplified. 

Y • ••' - : •* 
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Since the relationships between ideas in the sentenes becomes implicit, 
paradoxically the processing demands and inferencing skills required of_the 

- reader are increased (Kintsch & Vipond, 1979; Pearson, 1974-75) » Also, topic 
sentences are frequently deleted eliminating an integrative device for the 

^reader. As Davisi'on and Kantor (1982) "note, readability formulae not only 
don't define readability, but can be misused~sqch that they in fact decrease 

• the Comprehenslbility of text.~ „ •. . . 

Although there are numerous readability formulae available most use only a 

- few variables to determine the level of difficulty of a passage. These 
-variables occur at the word or sentence level and include: - a) word -frequency, 

«b) number of letters in a word,' c) number of syllables in a-word v d) number of 
words in a sentence, and e) number of sentences per passage (Klare, 1974-75). 
While researchers have examined other sentence characteristics such as 
prepositional phrases, degree of subordination, and passive verb forms, these 
variables have been less strongly correlated wi'th readability criteria. As a 
result, Word frequency, word length, and sentence length have remained the 
three variables most widely used in readability formulae due to strong 
predictive power and ease of use (Seldon; 1981). 

Tex*t Variables 

Cognitive Psychologists and Linguists have conducted numerous studies in 
which text variables hae been manipulated and the effects evaluated in terms 
of subsequent compreheni son performance.* 1 In particular, ;several text 
variables have been found to influence the comprehensibi lity of text. Kintsch 
and^Vipond (1979)- summarize the effects of these variables in their chapter 



A * ' " Text Variables^ 

r . • 

noting that the number of idea .t*nits in a passage, the number of different 
arguments- used in a text base, the number of explicit connectives- between 
sentences, -and the number of inferences required to connect the text 'base can 
increase .or decrease the processing demands made on the reader. Kintsch 
(1979) defines reading difficulty in terms of the meaning characteristics o'f 
the.text and the processes required of the reader to extract this meaning. He 
suggests that a formula of reading difficulty should include such variables 
as: numbers of reinstatement searches (the search of long term memory to 
match with inf ormation in the text) , word frequency,- idea unit density (the 
number of words per ideas expressed), the number of inferences, the number of' 
.processing cycles (cycles necessary to pick-up, match and store information in 
memory), and the number of different arguments (repeated ideas) in a texi 
base. Similarly, other researchers have found that the content structure of 
the text (Meyer, 1975) and the main^dea statements (Marshall & Clock, 
1978-79) can influence comprehension. Further, reader variables such as 
purpose for reading and or jJfi or .knowledge of the topic can compensate for 
poorly organized 'text and increase the amount of text processed (Kintsch &' 
Vipiond, 1979). 

For the purpose of thi s paper the following reader and text variables were 
considered for the evaluation of passage comprehensibil ity: content ? 
familiarity, vocabulary, the number ofidea units in the text, the number of 
text-basld inferences required to integrate ideas, the existence of main idea* 
statements, and the logical sequence of ideas in the passage. 



ERLC 



5 



Text Variables ' 

r 

5 



" °. ' METHODS - ' 

' *> * - w f, ^ • C i 

. . ; . ; ■ > • * > ; * ■ * •• 

Materials . * 

* ■ " ' .. . ■ ' ' • 

Three reading diagnostic -test's*, were selected for evaluation. , Criteria for 

selection Included recency, frequency of ui-e in the schools, and measurement 

- of comprehension performance. v The three tests selected were: .the Stanford* 

> 

Diagnostic Readi ng Test (SDftT), a grtfup test assessi ^ comprehension via 

, . " h 9 It - 4 

literal a rid"* Inferential questions; the Spathe DiagnostiVReading- Scales 
t (Spache) , an individually administered test also usfng questiGnSct.o,i$sess 
comprehension; and the Durrell Analysis of Reading Difficulty (Durrell), an 
individually administered test assessi ng comprehension 'vj^ a recall and probe 
. question technique. Selections from the secoid, fourth and sixth reader 
levels were evaluated to identify passage characteristics across reader levels 
and" teslr inconsistencies. ,. • 

Procedures . ■ * * . • 



With regard to text vari ables, each selection was transformed into a 
series 4 of propositions or idea units (Kintsch, 1974). In this manner, the 
number of idea units and text-based i nferences .required to.jntegrate ideas 
could be quantified for each passage. Also, ^ight advanced level graduate 
students rgted each selection with regard to content familiarity. Using a 
high, medium, low .scale, they rated each passage in terms of its relative 
familiarity to students whose grade placement corresponded to the reader level 
for which the passage was intended. 

• 0 



With regard to readabi lity. f ormul a vari ables, an apple computer program 
was employed, in which each passage was evaluated in terms of four readability 
formulae: the ARI, Fl esch-Kincaid, Har.ri s-Jacobson, and the Dale-Chall (see 
Klare, 1974-75, for a detailed description of these formulae). * 

• ''results and discussion 

Second Reader Level Passages * 

In an attempt to maintain a low readability index, passages wrtitten at the 
primary reader levels are typically very brief consisting of short,* disjointed 
simple sentences*and a simp! if ied vocaulary. These* passage features increase 
the processing demands for the reader, because concepts are vaguely defined 
(Langer, 1982) and the reader has to inf'ere'the misging information (Marshall, 
1979; Beck, McKeown, McCaslln & Burke, 1979). Thus, in an effort to reduce 
the readability of thes'e passages, the authors have, in fact, decreased their 

comprehens.ibility. *■ r 

^ ■ *.' 

As noted in Table 1, all three passages* were designed to assess second 

reader level comprehension skills. However, they vary considerably with 

regard to length (SDRT, 53 words and Spache, 119 words) apd readajnl i'ty . ; 

estimates within one passage (3.0 and 4.7 SDRT). This variability not only 

suggests that the passages are not comparable but .also calls into question the 

utility of readability indices designed to identify the reader Level of the 

passage. " 
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< **** • . . *> 

An examination of the text. variables contained in each of the passages . 

.reveals a few points worthy of attention. ' First-, .although, content familiarity 

is generally medium to- high across'all tiiree.selections* thereby facilitating 

■ '. "" ' '• ' ' ' 

comprehension,' the' number of .idea, units expressed in each passage varies 

considerably from 20 to 50 thus. al|eri ng the processing requirements Of ea£h 

of the tests. As Kintsch and Keen^n (1973) riot.e, an increase in the number of 

propositions, in a selection increases the processing time for that passage. 

Second, the. number of text-based inferences required to integrate,. ideas in the 

passages varies from 5 (DurreJl. and SORT) to 15 (Spache). Thus, even though 

all three Selections are designed. for the Seccnd reader level, the Spache \ 

requires, more text-abased inferences if comprehension" is to occur. Third", main 

idea statements 'are not present ip all three passages. In the absence o? such 

statements the reader must infer the gist "or main idea of the selection • 

thereby increasing processing, demands... A comparison of the three test 

passages reveals -that the Spache- provides a main/idea statement in the first 

. .. / 

sentence of;the paragraph." The JDurreVI does not contain such a statement; 

* 

however, it does have a title which could serve as an advanced organizer \ or 
the reader. .No main idea statement is present in the SDRT passage, making the 

■ * ' 

task of identifying the gist of this selection quite a challenge. In addition 

■ . \ 

the sentences of this paragraph do not appear to be logically related. The 

selection begins with the statement, "Bev-bara lived in a city. -..,«' then 
follows with,- "playing baseball in the park" and "eating lunch" and.closes 
with, "cows eat chocolate grass." ! ; 



Text Variables . 

Text organization can also. influence comprehensibi lity. Thre Spache 

selection appears to suffer from what Langer (,1982) calls imitation genre. 

* * ? 

That is, the text deviates from the expected. genre pattern and creates 

•confusion for the reader. In this selecton the first sentence is a main idea 

statement, ". ..Bob took a trip to the zoo. 1 : followed 'by sentences about what 

occurred when he first arrived at the zoo. The selection proceeds to a second 

paragraph which begins, "on t.he way out of the park..." completely omitting 

'information related to , what occurred during the visit and violating -the' = 

reader/s schema for the structure of the text (Meyer,. 1981) . This selection 

also contains^an illog.ical sentence Which*m.ight create confusion and 

comprehension difficulty for the reader. The sentence, "The cages were clean* 

but thelions didn't seem to like them..." leads the reader to believ| that 

lions like cages, if they are- cle"an, contrary to. what most readers have 

previously^learned. \ 

* In sum, all three test passages suffer from a lack of systematic control 

of text variables thus decreasing their comprehensibi lity. However, of the 

three "passages, evaluated at this level, the Spache appears to be the most 

difficult to comprehend because of its number of idea units, number of 

required i nf erences, , and imitation genre features. 

Fourth Reader Levei Passages 

Readability indices and text variables for passages at the fourth reader 
level~"are also listed in Table 1." In general, these passages differ from the 
second reader passages in terms of content familiarity, length, number of 



Text .Variables 

text-based inferences' and content structure. These selections are, most. 

difficult to comprehend because they are. written about less familiar topics 

(e.g. steamboats and bananas), they are longer, contain a greater number of 

idea units and required tex;t-base inferences per passage ^almost twice as 

many), and they are expository rather than narrative in nature (Meyer, 1981). 4 
~-- » " 

Consistent with, the findings at the second, reader level, the Spache 

• passage is almost twice as long as the other two selections. It has increased 
in length_.from 119 words to 216 words and in the number of idea units from 50 

, to 95. The Durrell selection has also almost doubled in length and'number of 
propbsitions while the SDRT passage has changed only moderately from 53 "to 88 
words and from 22 to 35 idea units. Tlie readability indices va>y across the 
passages from 4.9 'to 6.9, although they appear to fluctuate less within'a • 

- 1 : t» ; 

* ! 

selection than was noted previously perhaps due to the explicit nature of the 

passages at this level. . 

J . 
An examination of the' text variables in each of the passages also reveals 

observations consistent with those found "at" the second reader* 1 eve K The " 

Durrell passage-contains six" more - words than the SDRT, but both passages 

contain the same number of idea units* A compat^sori^of the number of 

inferences between passages reveals that although the Durell and SDRT contain 

the same number of idea units, , tlie Durrell requires a greater number of 

text-based inferences to link these -ideas together. These findings illustrate 

tbe--signif icant difference between readability variables such as passage 

length and text variables such' as idea units and text-based inferences which 

influence comprehensibility. 
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The pre-sence.xif main idea -statements in the passages continues' to vary at 
v " • '' ■ 

this level. TheDurrell passage contains a title arid a main idea statement in 

the first sentence of the paragraph. The selection is presented in logical 

oj-deji anj-^ontains explicit connectives which establish relationships between 

" ideas' in the 'sentences.- While the SDRJ did not provide a main idea statement 

at the second reader level, at this level it does have a main idea in the, form 

of a prequestion which serves Ms an excellent,, advanced organizer for ijfie 

reader. Perhaps the authors felt such a feature was. important to the 

comprehension of unfamiliar, expository text. The'Spache v selection does not 

contain a main idea statement and continues to lack strong text organization. 

Some of the sentences in this pAsage also contain cohesive ties which appear 

inappropriate and create further confusion for' the reader. For example, "He 

(Johft Paul's brother) wak eager to talk about the wonderful country, but John 

Paul already loved /vrierica^' or "Nothing unusua"f happened, but everyday was a 

■ ' / * \ i ~ ' 

real- adventure for the new ship's boy." | « . 

Even though all three passage's differ 'with, regard to the text variables, 
the Spache Selection-^ppars to have, the* greatest ^number of idea Units and ' * 
required text-based inferences,. To add to the difficulty of this passage, it • 
does not contain a title or main idea statement and is poorly structured, 
containing disjointed ideas and in appropriate 'connectives. TheDurrell and 
SDRT remain a bit more comprehensible and comparkble. 



Sixth Reader Level Passages "* \ 
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Table 1 contains the readability^ indices and tex>^ariables for passages 
a* 1 the sixth reader level. These selections are all expository in "nature and 
describe relatively unfamiliar topics.A fleas, building stone-, and the history 
of golf. ' \ 
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* 

An examination of surface. text features reve/als patterns similar -to those 
found at the other two reader levels. The test passages all vary with regard 
to length IBS' to 188 words) and readability i-ndices (from 5.4 to 9.12 reader 
levels), although readability measures "remain more stable for the Durrell and 
SDRT selections! Main idea placement patterns were similar across the fourth 
and sixth reader level passages. The Durrell 'selection contains a title which 
serves as its main idea and the SDRT contains a main, idea statement in the 

^N^st sentence. Conversely and consistently the Spache doesn't provide any , 

'•explicit. main idea statement. 

At this level some different text variables are present. The Durrell 
selection contains a few difficult vocabulary words such as "relented" and ■ 
"indulge;" however, both terms are defined implicitly by the contexj: of the 

/ . . f : • 

selection. This selection also cont^ns some less familiar expressions suph 
as, ''gained a foide following" arid "grown, in favor" as well as complex 
sentences thereby increasing the processing demands of the*text. However, 

\ 

this selection iswell structured to facilitate comprehension. 

The SDRT selection is similarly well organized. This' selection also uses , 
simile to facilitate comprehension of the unfamliar concept r^ silin : "...the 
resilin stretches like a rubber band when the flea is ready to jump." / * 

Ttie Spache selection consists of two paragraphs. This first is about a 
marble and the second about granite. Wh^le both are building stone, no" 4 
attempt is made to compare or contrast these stones to facilitate . « 

comprehension. Further, the organization of ideas within each paragaph is not 
parallel, resulting in increased difficulty for the reader in processing and 
remembering the information. Since no<rnain idea statement is provided, ,it ' 
will be difficult/for the , reader to identif.ythe relationship between these 
two types- of stonei 



Text Variables 
12 

In short, passages at the sixth reader level are similar to those at the 
fourth reader level with the exception that they contain slightly more ideas 
and begin to introduce less familiar vocabulary and complex 'sentence 
structures. ., Of the three test passages evaluated, the Spache appears to be 
less comprehensible due to the number of ideas presented and poor text 
organisation. . , . • • " 

i . ■ 

■«%».; " 

* p - " 

' ' - • • " ' 6 

Test Compari sons f • v 

~ ' r '.•■••«• 

Art •examinatjqp w»as made -of "the changes in number of idea units aijd . 

inferences contained in all the test passages from 'reader level two to six. v 

The. f indings revealed that on the, average the SDRT had the smallest number Vf 
. • • -a • « " . • • 

idea units and inferences when compared with the other'"two tests, and'the 

smallest increase in these variables from \level two to six (M=10.5 idea units, 

M=3 inferences). • The greatest number of idea units and inferences were found 

in the Spache selections along with*the greatest increase in these variables 

from level two to level six (M=27 i.dea units,\M=5 inferences). This summary 

data illustrates the differences in comprehensibility among these diagnostic 

tests. which in principle were designed to assess students 1 comprehension 

ability at approximately the sape reader levels. 
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Conclusion , . - • 

The three reading diagnostic tests evaluated all purport tcTassess 
compreheni son ability at the second-, fourth, and sixth reader levels.' 
However, this evaluation has shown that the three tests differ substantially 
from one another With regard to the number of ideas presented, the number of 
inferences necessary to integrate these ideas, and text organization 
features. Although readability indices have been employed to identify the/' 
difficulty level of ^hese passages, .these indices reflect only surface 
^features of theTtext and do not reflect the meaning characteristics of the 
• passage' nor identify the variables integral for deriving meaning from the 

text. is a result, these three tests contain features which decrease their 

'" ■ / '•■ ' i 

.comprehensibi 1 ity and confound the assessment of comprehension ability with 
'text comprehensibi 1 ity. 

Test authors have an obligation, to apply the results from prose ' » 
comprehension research and -systematically control text variables in test 
passages designed for reading diagnosis. -In the interim, however, teachers 
and diagnosticians use these tests regularly to' identify children with reading 
comprehension problems. Based upon this analysis, it is suggested that 
teachers become knowledgeable about those text variables which influence 
comprehensibi 1 ity. Tests which contain unfamiliar content, over-simplified or 
vaguely defined vocabulary, disjointed ideas, and poor text organization j 
"increase^ the processing demands of the reader and' are less comprehensible. 
Thus, if at all possible should not be used for diagnostic purposes. If tests 
which contain these features pave to be used due to state mandates or district 
-guidelines, teachers will want to evaluate these tests with regard to the text 
'variables discussed and consider to what extent a child has a" comprehension 
problem and to what extent the text "is poorly constructed thereby creating 
comprehension difficulties "for the student. 
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Readability Indices and Text Variables in Second, Fourth and Sixth Reader Level 
V * • Passages of Three Reading Diagnostic Tests. * ■ / ' 
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- 1 




• 


TESTS 


C 


- 






Variables 

• 




Durrell 


• 




Spache 






SORT 




# Words 


a 54 


b 94 


c 128 


a 119 * 


Si 6 


C 188 


a 53 


b 88 


c 88 


# Sentences 


> 5 


6 


7 *' 




16 


17 




6 


5 

v 


, # Syllables 


67.5. 


131.1 


T87 


M47.5 r 


292.4 


- 273.7 


75.8 


114 


124.7 


Readability: 




«* " 




/' 




•a. 




> 




ARI 


2.3 


6.8 


9.1- 


3.3 


5.2 


° .5.4 


3.9 


4.9 




j * Flesch-Kincaid 


3.4 


6.9 


8,8 


.4,'2 ' 


5.6 


5.9 


.4.7 


5,4 ' 


7.9: 

Y - . 


1 Harris-Jacobson 


2.2 




. ... 


3 .2 






3 




Dale-Chall 


below 

;:" 4th 


. '.5-6 


7-8 


below 
4tn 


5-6 


7-8 


below 

A + V\ 


4th 6r 1 

less 


* 7-8 . 

p * 


Content Familiarity* 


med-high med-low 


med-low 


high 


low 


low-rced 


' high 


med . 


*. v low , 


#' Idea Units 


' ,20 


35 


56 


50 


95 


104 

0 


22 


'35 


43 


.. # Text Based Inferences 




11 


16 


15 


27 

t 


25 , 


5 


7 


11 


Main Idea Statement 


title 


title 
first 
sentence 


title 


first 
sentence 








first 
i sentence 


firist t '.} 
sentence 


a Second Reader Level 






V 




o 

a 

1 










• Fourth Reader Level 
o 

■ EWCixth Reader Level 




/ 

1 7 




O 










18 ; .'• 



